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Abstract Mind Architecture 
(in artificial intelligence) 


Two Processes 


Perception/World Modeling: 


Vision, sensor processing, 
sensor fusion, ... 


Action Selection/Decision Making: 


Planning, plan execution, 
goal deliberation, reacting... 


My Scientific Problem: 
The Nature of the Social Mind 


The computational mechanisms that allow the mind to 
reason about, and interact with, others? 


(differently than about or with inanimate objects) 


Abstract Mind Architecture 


(in artificial intelligence) 


Two Processes 


Perception/World Modeling: Action Selection/Decision Making: 


Theory of Mind: Coordination: 


Intents, plans of other minds Manipulating, acting 
(in AI: Plan Recognition) w.r.t other minds 


\ This talk 


Since 1995.... 


Tracking teammates (RESL, w/ Tambe) 

Overhearing teams (YOYO, YOYO*, w/ Tambe, Pynadath) 
Overhearing using Colored Petri-nets (w/ Gutnik) 
Symbolic Recognition (SBR, w/ Avrahami-Zilberbrandt) 
Decision-theoretic rec. (UPR, w/ Avrahami-Zilberbrandt) 
Intent detection (w/ Bochek-Dokow) 

Mirroring (w/ Vered) 


See: www.cs.biu.ac.il/-galk/publications/class_rescat.html 
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Recognition Challenges 


Online: Observations incrementally received 
Complex state observations 
— Observe state, not agent actions 
— State is factored (composed of multiple features) 
Situated (reactive) 
— Observed agents react, deviate from own plans 
— What goal/plan is true now 
Observations stem from continuous world 
— Sensitivity to discretization (loss of information) 


SBR: Symbolic Behavior Recognition 
[IJCAI 2005, MOO 2004-2005] 


* Online, keyhole recognition, discrete observations 
* Applied to vision tracker data (e.g., airport security) 


SBR: Symbolic Behavior Recognition 
[UCAI 2005, MOO 2004-2005] 


* Online, keyhole recognition, discrete observations 
٠ Applied to vision tracker data (e.g., airport security) 


What is agent doing? 


Is its behavior anomalous? 


SBR Plan Library: Layered, Durative Actions 


Directed acyclic connected graph 


security 


Vertices denote plan steps (actions) 
Edges 
— Vertical (decomposition) edges 
— Horizontal (sequential) edges 


Every vertex may generate obs, have duration (cycles) 
“Or”-graph (edges denote ordered choices) 


Example: Plans and Hypotheses 
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Example: Plans and Hypotheses 


without 
bag 


Hypotheses NOT matching obs “look” 
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SBR Key Ideas 


e Sacrifice memory to gain speed 
— Entire grounded library in Memory, in advance 
— Auxiliary data structures to support fast queries 
e Distinguish recognition queries 
— Given last observation, what might be true now 
— Given history of observations, what might have been true 
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e Sacrifice memory to gain speed 
— Entire grounded library in Memory, in advance 
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e Distinguish recognition queries 
— Given last observation, what might be true now 
— Given history of observations, what might have been tru 


Thi 1 
Is allows computing only what is needed! 


» (6) 


Current State Query 
What is True 


* Generating plan-path hypotheses: 
— Tag matching plans by observation time 
— Propagate tags up/down according to temporal consistenc 


* Plan P is temporally consistent in time-stamp t if: 


— No incoming sequential edges 


— OR: Exist previous plan tagged with SEA © O 
— OR: Tagged with t-1 EM 


e Efficient: O(M log L) 


— M number of matching plan steps QOL 


— Lsize of library 


Current State Query: Example 
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Current State Query: Example 


Time 1: Observation is “look” 
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Current State Query: Example 
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Current State Query: Example 
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Time 2: Observation is "moving" 
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Current State Query: Example 


entrance 
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Current State Query: Example 
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Current State Query: Example 


entrance 
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Time 3: Observation is “gate” 


TO 


Current State Query: Example 


History of States Query 
What was True 


٠ CSQ is situated: only hypothesizes as to current state 


* Does not explain complete sequences 
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AII possible sequences (1,2,3)? No. 


History of States 


x-ray completes security 
Security terminates at time 2 . (4) 


History of States 


Security must start at time 1 
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History of States 


History of States Query 
* Generate (or incrementally build) hypotheses graph 


— Vertices: denoting hypotheses at time t 
— Edges: connect valid continuations from t to t+1 


Security look 


Security Security > 
xray> with xray > 
bag without bag 


Board> gate 


1 Entrance > 
look 


Entrance> Entrance> 
2 |move2 move > 


Board 
move 


with bag without bag 


History of States Query 


e Valid hypotheses: paths from vertices in time t to time 1 


Time 


1 Entrance > 
look 


Security look 


Security > 
xray > 
without bag 


Board— gate 


Entrance> Entrance> 
2 | move > move > 


Security 
xray» with 
bag 


Board 
move 


with bag without bag 


SBR Highlights 
[UCAI 2005, MOO 2004-2005] 


* Highly efficient, Complete: 


— Match observations to plan library vertices in O(1) 
— CSQ is O(M log L), M # of matches, L size of library 
— History query is polynomial: 
e Graph construction is O(M?) for each observation 
* Hypotheses extraction is O(TM?) 
e Extensions for interleaving, interrupting, ... 


* Limitation: grounded library, must fit in memory 


Lazy Commitment tn SBR 


٠ Book keeping allows delaying inference 
° Compute only if queried 

— e.g, hypotheses graph can be built only on history query 
* No commitment to ranking 


— Probabilistic or decision-theoretic ranking is separate 
— SBR as filter [AAAI 2007]; PHATT, SLIM use similar approach 


Mirroring 
[ACS 2016, IJCAI 2017, AAAI 2018] 


Observations and Plans are in Continuous Space 


gestures, motions, goal locations: trajectories 
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Discretization (e.g., using grid) 


Discretization (e.g., using grid) 
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Resulting Observations 
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Early Commitment to Discretization 


Theorem 1: 
For any discretization, can find case where it fails 


What is the goal of the agent? 
A or B? 


n, 


Continuous Version Discrete Version 


Late Commitment to Discretization 


Theorem 2: 
For any case, can find discretization where it succeeds 


What is the goal of the agent? 
A or B? 


n, 


Continuous Version Discrete Version 


Challenge: 
Allow Late Commitment to Discretization 


٠ Plan recognition libraries require early commitment 
— Hierarchical structures, grammars, HMMSs and variants 


* Plan recognition by planning (PRP) has potential 
— Generates hypotheses ad-hoc, after getting observations 
— Expensive for online recognition 


PRP in Continuous Domains 


— Observations are of continuous actions? 
— Requires domain theory describing continuous domains 


— Requires planner that can work in continuous domains 
— Lots of these in OMPL (Open Motion Planning Lib) 
— But cannot compute path that “deviates from O” 


Mirroring [IJCAI 17, AAAI 18] 


° Revised procedure for ranking hypotheses 
— Optimal plan (plan(G)) vs observed plan (plan(G+0)) 
— Closer to formulation in [R&G 2009] (abandoned?) 

° Generalize planning domain theories 
— Actions generate trajectories, not single final state 

e Online recognition 
— 2T planner calls, each goal Y (T+1) planner calls each 
— Heuristics can improve performance (guaranteed) 


Plan Recognition Problems 


e Given R = <W,1,G,O> 
— W domain; | initial state; G set of possible goals; O observations 
° Find plans in W, from to a goal in G, that match O 


2D example 


| | Action 
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Action Action 


Action 


Initial State 


Plans in Domain W 


Plan: sequence of actions 
٠ States defined using fluents (numerical values allowed) 
— e.g., on(A,B)=true, fuel-remaining(robot)=50.33, pos-x(r)=4.5 


e Actions: trajectories of state-changes 
— 6(s, 5a) = (Spears 
- 5 = (S \DEL,(S)) © ADD (S) 


— Proper generalization of STRIPS actions 
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Finding good plan hypotheses 


e Want: plan hypothesis mt, = argmax P(rt|O) 
— Intuitively - “best matches the observations” 


* We want matching that maximizes 
P(n|O) =BP(Oln) Pin) 
= B P(O|n) P(n|g) P(g) 
ی‎ == 
لبا‎ Plan, sr sù 


Obs., given plan 


Focus: Maximize P(O |r), P(rt|g) 


How to maximize P(O|7), P(rt|g): 


Two principles: 
e P(OIn) [obs. given plan]: prefer plans matching obs. 


— Minimize Error(n, 0): Accumulated distance between m and O 
— P(O|m) = 1 / (1+Error(rt, O)) 


٠ P(rt|g) [plan given goal]: prefer optimal plans 
— Assume rationality of observed 


— Higher P(n|g) when x closer to ideal plan x 
— Defined as (normalized) ratio between costs of rt and rt” 


NO 


How to maximize P(Olr), P(zt| g): 
Shortcut 


Two principles: 
e P(OIn) [obs. given plan]: prefer plans matching obs. 


— Minimize Error(n, O): Accumulated distance between m and O 
— P(O|n) = 1 / (1+Error(rt, O)) 


= 1 when perfect F plan that goes through obs 
٠ P(rt|g) [plan given goal]: prefer optimal plans 


— Assume rationality of observed Calls to planner 


— Higher P(r|g) when x closer ta ideal plan 7c 


— Defined as (normalized) ratio between costs of rt and rt” 
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Domains 


٠ Motion planners in OMPL [Sucan et al. 2012] 
e Polygon drawing planner [Vered et al. 2016] 
٠ ROS MoveBase standard navigation package 


Summary: Lazy Is Good 


Late commitment to discretization: a MUST 

— Remember the theorems (and shown experimentally) 

— Mirroring: a novel form of PRP allowed this 

Late commitment to computing queries: Efficient 
Promising: distinguish queries in PRP 

— Ignore (most of) the past [Masters & Sardina 2017] 

— Pre-computing [Marting et al. 2015, Pereira et al. 2016, 2017] 


Thanks: Friendly organizers and atmosphere at PAIR 


galk@cs.biu.ac.il 
http://www.cs.biu.ac.il/~galk/ 
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